Skip to content

Use specific failure categories before agent_failure in failure-issue routing#44498

Merged
pelikhan merged 2 commits into
mainfrom
copilot/classify-failure-causes
Jul 9, 2026
Merged

Use specific failure categories before agent_failure in failure-issue routing#44498
pelikhan merged 2 commits into
mainfrom
copilot/classify-failure-causes

Conversation

Copilot AI commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

report-failure-as-issue was treating many non-agent failures as agent_failure because generic classification was always added on job failure. That made category filters noisy, especially for infra/transient failure modes where existing specific categories already exist.

  • Failure classification tightening

    • Updated buildFailureMatchCategories(...) in handle_agent_failure.cjs to emit agent_failure only as a fallback.
    • agent_failure is now added only when:
      • conclusion is failure,
      • not timed out, and
      • no specific category has already been detected.
  • Category-filter behavior alignment

    • This preserves existing specific categories (for example inference_access_error, report_incomplete, awf_firewall_startup_failed) as the primary signal used by report-failure-as-issue include/exclude filters.
    • Result: include filters like ["agent_failure"] no longer match infra-classified failures just because the job failed.
  • Targeted test updates

    • Added/updated tests to assert:
      • agent_failure is omitted when specific categories are present.
      • infra-classified failures are skipped by include-only ["agent_failure"] filters.
// Before: generic category added alongside specific categories
if (options.agentConclusion === "failure" && !options.isTimedOut) {
  categories.push("agent_failure");
}

// After: generic category only when no specific classification exists
const hasSpecificFailureCategory = categories.length > 0;
if (options.agentConclusion === "failure" && !options.isTimedOut && !hasSpecificFailureCategory) {
  categories.push("agent_failure");
}

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Copilot AI changed the title [WIP] Add classification of failure causes before reporting issues Use specific failure categories before agent_failure in failure-issue routing Jul 9, 2026
Copilot AI requested a review from pelikhan July 9, 2026 05:56
@pelikhan
pelikhan marked this pull request as ready for review July 9, 2026 10:55
Copilot AI review requested due to automatic review settings July 9, 2026 10:55

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This pull request tightens failure-category classification in the report-failure-as-issue routing logic so that agent_failure is emitted only as a true fallback when no more specific failure category was detected, reducing noise in category-based include/exclude filters.

Changes:

  • Updated buildFailureMatchCategories(...) to add agent_failure only when the job failed, did not time out, and no specific failure category has been detected.
  • Added/updated tests to assert agent_failure is omitted when a specific category (e.g., report_incomplete, awf_firewall_startup_failed) exists.
  • Extended filter-behavior tests to ensure infra-classified failures (e.g., inference_access_error) do not match include-only ["agent_failure"] filters.
Show a summary per file
File Description
actions/setup/js/handle_agent_failure.cjs Makes agent_failure a fallback-only category when no other specific failure category has been detected.
actions/setup/js/handle_agent_failure.test.cjs Adds assertions covering fallback-only behavior and category-filter routing with infra-classified failures.

Review details

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

  • Files reviewed: 2/2 changed files
  • Comments generated: 0
  • Review effort level: Low

@github-actions

github-actions Bot commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

Test Quality Sentinel failed during test quality analysis.

@github-actions

github-actions Bot commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

⚠️ PR Code Quality Reviewer failed during code quality review.

@github-actions

github-actions Bot commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

Design Decision Gate 🏗️ failed during design decision gate check.

@pelikhan
pelikhan merged commit b143f58 into main Jul 9, 2026
55 of 69 checks passed
@pelikhan
pelikhan deleted the copilot/classify-failure-causes branch July 9, 2026 11:55
@github-actions

Copy link
Copy Markdown
Contributor

🎉 This pull request is included in a new release.

Release: v0.82.8

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Feature: classify failure causes (infra vs agent) before eport-failure-as-issue triggers

3 participants